Image acquisition method and apparatus

ABSTRACT

An image acquisition method operates in a hand held image acquisition device with a camera. A first image of a scene is obtained with the camera at a nominal exposure level. A number of relatively bright pixels and a number of relatively dark pixels within the first image are determined. Based on the number of relatively bright pixels, a negative exposure adjustment is determined and based on the number of relatively dark pixels, a positive exposure adjustment is determined. Respective images are acquired at the nominal exposure level; with the negative exposure adjustment; and with the positive exposure adjustment as component images for high dynamic range (HDR) image of the scene.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a U.S. Non-Provisional Patent Applicationclaiming the benefit of priority of U.S. Provisional Patent ApplicationNo. 62/147,464, filed on Apr. 14, 2015, the content of which isexpressly incorporated by reference herein in its entirety.

FIELD

The present invention relates to an image acquisition method andapparatus.

BACKGROUND

High-dynamic-range (HDR) photographs are typically generated byacquiring multiple component images of a scene, each with differentexposure levels, and then later, merging the component images into asingle HDR image.

Some prior art approaches can involve acquiring a set of, for example, 8component images, across a range of different exposure levels to ensurethat appropriately exposed image information is available for allregions of a HDR image. However, this can involve significant delay inacquiring the component images and so can result in blurring or ghostingartefacts from one image to the next. It can also require a large amountof processing to handle the set of component images.

U.S. Pat. No. 8,724,921, Aptina, discloses a method for capturing a highdynamic range (HDR) image. Multiple component images of a scene arecaptured at respectively different exposure settings. A further image ofan object placed in the scene is captured at one exposure setting. Afirst radiance image is formed from the multiple component images. Asecond radiance image is formed from the further image. The firstradiance image and the second radiance image are merged to form the HDRimage.

Natalia Gurieva “Complete Digital Workflow for HDR Photography”,International Circular of Graphic Education and Research, No. 7, 2014,pp 14-23 discloses capturing a HDR image including conducting a dynamicrange evaluation of the scene. Depending on the type of the scene andits dynamic range, different capturing strategies can be applied. Forexample, in case of midday sun with strong shadows it will be enough totake 3 shots at about 1 to about 1.33 stops apart; inside buildings withsome light coming through the windows at least 5 bracketed shots atabout 2 stops apart have to be taken.

U.S. Pat. No. 8,687,087, CSR Tech discloses a different approach tocapturing an image of a scene which would require high dynamic rangeincluding adjusting exposure time on an image block by block basis tomaintain a resulting digital signal within a range carried by a digitalprocessing path that carries a limited number of bits.

US 2014/0002694, CSR Tech discloses capturing two or more image framesusing different exposure settings and then combining the images to forma single HDR output frame in a video sequence. A pipelined architectureoperates on adjacent image frames by performing image alignment, imagemixing and tone mapping on the adjacent image frames to generate the HDRimage sequence.

It is an object of the present invention to provide an efficienttechnique for determining appropriate exposure levels for a limitednumber of component images used to provide a HDR image.

SUMMARY

According to a first aspect of the present invention there is providedan image acquisition method disclosed herein.

In still further aspects there is provided an image acquisition devicearranged to perform the methods disclosed herein.

Separately, there is provided a non-transient computer readable mediumcomprising computer executable instructions, which instructions whenexecuted on an image acquisition device, cause the image acquisitiondevice to perform the methods disclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention will now be described, by way ofexample, with reference to the accompanying drawings, in which:

FIG. 1 is a block diagram of an image acquisition device on whichembodiments of the present invention can be implemented;

FIG. 2 is a flow diagram illustrating an image acquisition methodoperable on a device such as shown in FIG. 1;

FIG. 3(a) illustrate thresholding curves employed in the method of FIG.2;

FIG. 3(b) shows exemplary image data combined with the thresholdingcurves of FIG. 3(b); and

FIG. 3(c) illustrates exposure value conversion curves employed in themethod of FIG. 2.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring now to FIG. 1 which is a block diagram of an image acquisitiondevice 20, which in the present embodiment is a portable digital camera,operating in accordance with certain embodiments. It will be appreciatedthat many of the processes implemented in the digital camera areimplemented in or controlled by software operating on a microprocessor,central processing unit, controller, digital signal processor and/or anapplication specific integrated circuit, collectively depicted asprocessor 120. All user interface and control of peripheral componentssuch as buttons and display is controlled by a microcontroller 122.

In operation, the processor 120, in response to a user input at 122,such as half pressing a shutter button (pre-capture mode 32), initiatesand controls the digital photographic process. Ambient light exposurecan be determined using a light sensor 40 in order to automaticallydetermine an exposure setting for image acquisition. The distance to thesubject is determined using a focusing mechanism 50 which also focusesthe image on an image capture device 60. In a flash mode of the device,the processor 120 can cause a flash device 70 to generate a photographicflash in substantial coincidence with the recording of the image by theimage capture device 60 upon full depression of the shutter button.Flash mode may be selectively generated either in response to the lightsensor 40 or a manual input from the user of the camera.

The image capture device 60 digitally records acquired images in colour.The image capture device is known to those familiar with the art and mayinclude a CCD (charge coupled device) or CMOS to facilitate digitalrecording. High resolution images recorded by image capture device 60 isstored in an image store 80 which may comprise computer memory such adynamic random access memory or a non-volatile memory. The camera isequipped with a display 100, such as an LCD, both for displaying previewimages and displaying a user interface for camera control software.

In the case of preview images which are generated in the pre-capturemode 32 with the shutter button half-pressed, the display 100 can assistthe user in composing the image, as well as being used to determinefocusing and exposure. Temporary storage 82 is used to store one orplurality of the stream of preview images and can be part of the imagestore 80 or a separate component. The preview image is usually generatedby the image capture device 60. For speed and memory efficiency reasons,preview images may have a lower pixel resolution than the main imagetaken when the shutter button is fully depressed, and can be generatedby sub-sampling a raw captured image using software 124 which can bepart of the general processor 120 or dedicated hardware or combinationthereof.

In the present embodiment, a face detection and tracking module 130 suchas described for example, PCT Publication No. WO2008/018887 (Ref:FN-143), is operably connected to the sub-sampler 124 to control thesub-sampled resolution of the preview images in accordance with therequirements of the face detection and tracking module. Preview imagesstored in temporary storage 82 are available to the module 130 whichrecords the locations of faces tracked and detected in the preview imagestream. In one embodiment, the module 130 is operably connected to thedisplay 100 so that boundaries of detected and tracked face regions canbe superimposed on the display around the faces during preview.

In FIG. 1, the face tracking module 130 is arranged to extract and storetracked facial regions at relatively low resolution in a memory buffersuch as memory 82 and possibly for storage as meta-data in an acquiredimage header stored in memory 80. Where multiple face regions aretracked, a buffer is established for each tracked face region. Thesebuffers are of finite size (10-20 extracted face regions in a preferredembodiment) and generally operate on a first-in-first-out (FIFO) basis.

The device 20 can further include an image correction module 90 whichmay perform post processing of any acquired images. Where the module 90is arranged for off-line correction of acquired images in an externalprocessing device 10, such as a desktop computer, a colour printer or aphoto kiosk, face regions detected and/or tracked in preview images arepreferably stored as meta-data within the image header. However, wherethe module 90 is implemented within the camera 20, it can have directaccess to the buffer 82 where preview images and/or face regioninformation is stored.

The module 90 can thus receive the captured high resolution digitalimages from the store 80 and analyze these to improve the quality of theimage. The module can modify the image and the modified image may beeither displayed on image display 100, saved on a persistent storage 112which can be internal or a removable storage such as CF card, SD card orthe like, or downloaded to another device via image output means 110which can be tethered or wireless. The module 90 can be brought intooperation either automatically each time an image is captured, or uponuser demand via input 30. Although illustrated as a separate item, wherethe module 90 is part of the camera, it may be implemented by suitablesoftware on the processor 120.

In embodiments of the present invention, the image acquisition deviceprovides a HDR mode 72 selectable by the user of the image acquisitiondevice 20. Typically, HDR mode is chosen as an alternative to anon-flash automatic exposure mode; or a flash exposure mode; or indeedany number of other specialised image acquisition modes such as portraitor panorama modes.

Referring to FIG. 2, the mode is typically implemented by the processor120 as follows: An image of a scene is captured at a nominal exposurelevel (0 EV), step 200. The 0 EV exposure level for this image can bedetermined based on an average light level of a scene provided by thelight sensor 40; or can be determined based on spot metering of thescene; or can be determined based on a exposure level of for example,one or more face regions detected within the scene by the face tracker130.

The 0 EV image can be a preview image and displayed on the image display100 as normal. Once the 0 EV image is acquired, it is analysed in step202 to determine the exposure levels for the remaining component imagesto be used in producing a HDR image.

This analysis first of all comprises computing a histogram of pixelintensities for the 0 EV image. It will be appreciated that thistechnique lends itself to processing images in formats where intensityis provided in a separate image plane. Nonetheless, images can beacquired in any number of formats including RGB, LAB, YCC, YUV etc. Insome cases such as LAB, YCC or YUV, one of the colour planes L or Yprovides an intensity value 0≤Intensity≤Maxlevel, typically 255 for 8bit pixels, directly, whereas for others such as RGB, colour planeinformation needs to be combined or transformed to provide an intensityvalue, for example, Intensity=0.2126*R+0.7152*G+0.0722*B.

Referring now to FIG. 3(b), a histogram for an image (histlmg) acquiredfrom a typical scene is shown.

The analysis now uses respective dark and light thresholding (orweighting) curves (curveDark, curveLight), FIG. 3(a), each curve rangingin value from 0 to 1, to obtain a measure of the extent of dark andburned areas within the image.

In the example, the dark thresholding curve (curveDark) is zero valuedabove a relatively low corner intensity value (DT) and increases invalue towards 1 at zero intensity. Conversely, the light thresholdingcurve (curveLight) is zero valued below a relatively high cornerintensity value (LT) and increases in value towards 1 at maximumintensity (in this case 255). Each curve increases monotonically with afirst low inflection point and a second higher inflection point, withcurve values tending to plateau as they increase towards 1. As such,each of curveDark and curveLight can be thought of as low pass and highpass filters for image histogram information.

The thresholding curves are based on the observation that in a wellilluminated scene most pixel intensity values are the middle of therange. Some well illuminated scenes can still include dark and lightpixels, but these should be relatively low in frequency, and need notnecessarily cause a dynamic range problem.

If DT and LT alone were used as simple thresholding values fordetermining the number of dark and bright pixels in an image and toprovide indicators of the relative darkness or lightness of an image,then exposure adjustments for images including normally occurring darkand bright pixels would tend to be too great. The thresholding curves ofFIG. 3(a) therefore tend to limit the effect of most bright and mostdark pixels on the exposure adjustments which may be made for additionalcomponent images.

By multiplying the values for histlmg against curveDark and curveLightas follows:darkPrc=W _(D)*Σ_(i=0) ²⁵⁵(histImg_(i)*curveDark_(i))lightPrc=W _(L)*Σ_(i=0) ²⁵⁵(histImg_(i)*curveLight_(i))respective measures of the extent of dark areas within the image(indicated as Dark Areas in FIG. 3(b)) and light areas within the image(indicated as Burned Areas in FIG. 3(b)) can be provided. In theexample, a weight W_(D)=W_(L)=100 is used to scale the summed values andcan be dependent on the profile of the thresholding curves. For exampleW_(D)*∫curveDark=W_(L)*∫curveLight=1. In the example of FIG. 3(b), theregion indicated as “dark areas” illustrates the product of histlmg andcurveDark; while the region indicated as “burned areas” illustrates theproduct of histlmg and curveLight. The values for these regions aresummed (integrated) and scaled to provide darkPrc, lightPrc.

In the example, of FIG. 3(b), burned areas indicates a greater amount ofsaturated image area than dark areas.

These resultant values darkPrc and lightPrc are now used as lookupvalues to determine an initial plusEV and minusEV adjustment for twocomponent images to be subsequently acquired as follows:minusEV=−minusCurve(lightPrc);plusEV=plusCurve(darkPrc);

The values within the lookup table are determined experimentally andincrease in proportion to the darkPrc and lightPrc figures respectively.It will be noted that for a given area of dark pixels (darkPrc), theabsolute value of the overexposure adjustment level (plusEV) will beless than the absolute value of the underexposure adjustment level(minusEV) for the same area of bright pixels (lightPrc).

Referring back to FIG. 2, in the example, a minusEV of −0.7 and a plusEVof 0.5 are determined for an 0 EV image. Exposure settings are adjustedaccordingly and a respective preview image can be captured at each ofthese settings 204 a, 204 b; or indeed a relatively short exposure time(SET) image can be acquired at the minusEV setting and exposure allowedto continue without clearing the sensor before acquiring a longerexposure time (LET) image at the plusEV setting.

As an alternative or in addition to adjusting exposure time, if theimage acquisition device 20 comprises an adjustable aperture, thenaperture stop can also be adjusted to adjust the exposure of the imagescaptured in steps 204 a and 204 b. However, in this case such imageswould have to be acquired successively.

Each of the SET and LET images are then analysed in the same manner asthe 0 EV image in steps 202 a and 202 b, except in the case of the SETimage, only lightPrc is of concern; whereas for the LET image, onlydarkPrc is of concern.

If the analysis 202 a of the SET image provides lightPrc with anabsolute value>0, then the minusEV value is adjusted further, step 204aa.

Similarly, if the analysis 202 b of the SET image provides darkPrc withan absolute value>0, then the plusEV value is adjusted further, step 204bb.

In the example, the final minusEV and plusEV values are chosen as −1.2EV and +0.8 EV respectively.

Now each of a final minusEV and plusEV image are acquired at steps 200 aand 200 b respectively. Again, the minusEV image can be acquired at theminusEV setting and exposure allowed to continue without clearing thesensor before acquiring the longer exposure time plusEV image at theplusEV setting. In this case, each of the minusEV, a second 0 EV andplusEV images can be acquired successively from the sensor to mitigateproblems with blur or ghosting between the minusEV and plusEV images andthe 0 EV image acquired at step 200.

Again, as an alternative or in addition to adjusting exposure time, ifthe image acquisition device 20 comprises an adjustable aperture, thenaperture stop can also be adjusted to adjust the exposure of the imagescaptured in steps 200 a and 200 b. Again, in this case such images wouldhave to be acquired successively.

In this case, where the 0 EV image acquired at step 200 is not used as acomponent image in HDR processing, a full resolution image need not becaptured at step 200.

Where the minusEV and plusEV images are acquired separately, each of the0 EV and these images may need to be aligned before they can be combinedinto a HDR image and there can be problems with motion blur where theseis movement within a scene.

On the other hand, where the minusEV and plusEV and possibly 0 EV imagesare acquired substantially contemporaneously, ghosting artefacts canarise and special processing may be required to deal with theseartefacts.

In any case, once the 0 EV, minusEV and plusEV component images areavailable, they can be combined at step 206 into a HDR image. Thisprocessing can be performed immediately so that the HDR image can beviewed immediately on the acquisition device 20, or for example, theprocessing can be performed with a post processing module such as thecorrection module 90.

It will be appreciated that where the relatively short exposure time andrelatively long exposure time images acquired in steps 204 a and 204 bare full resolution images, these could also be retained for use ascomponent images of a HDR image.

It will be appreciated that each of the analysis steps 202, 202 a and202 b need not analyse every pixel of an input image. So for example, ifa full-resolution preview image has been acquired, it can be sub-sampledfor the purposes of determining minusEV and plusEV. In any case, it willbe seen that the above technique enables an image acquisition device toquickly acquire a minimal set of HDR images at suitable exposuresettings for subsequent use in creating a HDR image.

The invention claimed is:
 1. An image acquisition method operable in ahand held image acquisition device with a camera comprising: a)obtaining a first image of a scene with the camera at a nominal exposurelevel; b) determining a number of relatively bright pixels within saidfirst image; c) determining a number of relatively dark pixels withinsaid first image; d) determining based on said number of relativelybright pixels a negative exposure adjustment; e) determining based onsaid number of relatively dark pixels a positive exposure adjustment;and f) acquiring respective images at said nominal exposure level, withsaid negative exposure adjustment and with said positive exposureadjustment as component images for a high dynamic range (HDR) image ofthe scene, wherein: said determining said negative exposure adjustmentfurther comprises: acquiring a second image of said scene with anegative exposure adjustment based on the number of relatively brightpixels within said first image; determining a second number ofrelatively bright pixels with said second image; and further adjustingsaid negative exposure adjustment as a function of the second number ofrelatively bright pixels; or said determining said positive exposureadjustment further comprises: acquiring a third image of said scene witha positive exposure adjustment based on the number of relatively darkpixels within said first image; determining a third number of relativelydark pixels with said third image; and further adjusting said positiveexposure adjustment as a function of the third number of relatively darkpixels.
 2. A method according to claim 1 further comprising generating aHDR image based on combination of said component images.
 3. A methodaccording to claim 1 wherein said obtaining said first image comprisesobtaining a preview image of the scene.
 4. A method according to claim 1wherein said obtaining said first image comprises sub-sampling an imageof the scene.
 5. A method according to claim 1 wherein said second imageis one of a preview image or a sub-sampled image.
 6. A method accordingto claim 1 wherein said third image is one of a preview image or asub-sampled image.
 7. A method according to claim 1 comprising acquiringsaid component images substantially contemporaneously.
 8. A methodaccording to claim 1 comprising successively acquiring said componentimages.
 9. An image acquisition device arranged to perform the method ofclaim
 1. 10. A non-transient computer readable medium comprisingcomputer executable instructions, which instructions when executed on animage acquisition device, cause the image acquisition device to performthe method of claim
 1. 11. An image acquisition method operable in ahand held image acquisition device with a camera comprising: a)obtaining a first image of a scene with the camera at a nominal exposurelevel; b) determining a number of relatively bright pixels within saidfirst image; c) determining a number of relatively dark pixels withinsaid first image; d) determining based on said number of relativelybright pixels a negative exposure adjustment; e) determining based onsaid number of relatively dark pixels a positive exposure adjustment; f)acquiring respective images at said nominal exposure level, with saidnegative exposure adjustment and with said positive exposure adjustmentas component images for a high dynamic range (HDR) image of the scene,and g) providing a histogram of pixel intensities for said first imagewherein: said determining a number of dark pixels comprises: low passfiltering said histogram and integrating said low pass filteredhistogram; and a corner frequency of said low pass filtering is lessthan an intensity of 50 in a range from 0 to 255; or said determining anumber of bright pixels comprises: high pass filtering said histogramand integrating said high pass filtered histogram; and a cornerfrequency of said high pass filter is more than an intensity of 200 in arange from 0 to
 255. 12. A method according to claim 11 furthercomprising generating a HDR image based on combination of said componentimages.
 13. A method according to claim 11 wherein said obtaining saidfirst image comprises obtaining a preview image of the scene.
 14. Amethod according to claim 11 wherein said obtaining said first imagecomprises sub-sampling an image of the scene.
 15. A method according toclaim 11 comprising acquiring said component images substantiallycontemporaneously.
 16. A method according to claim 11 comprisingsuccessively acquiring said component images.
 17. An image acquisitiondevice arranged to perform the method of claim
 11. 18. A non-transientcomputer readable medium comprising computer executable instructions,which instructions when executed on an image acquisition device, causethe image acquisition device to perform the method of claim 11.